Data Led Growth project | The Kanji Company

Overview

Objective Setting

This is an attempt to design a suite of experiments, the results of which will give us insights into how to build out the strategy in the coming two months to primarily reduce the Customer Acquisition Cost (CAC) and secondarily increase the Org's Advertisement Click Through Rate (CTR) on Instagram.

The scenario which we will address is as follows - The Org has been running a New Content Strategy, we want to understand how it is performing by analyzing October'24 numbers against September'24 numbers.

What is the Product?

The product is a nostalgic fermented drink called 'Kanji', which many people in northern parts of India might remember from their childhood. Kanji is fermented drink made by fermenting ground mustard seeds in a solution of water, salt and hints of asafoetida (Heeng). Kanji also has carrot, radish and beetroot incorporated in diced form to enhance the flavor and color of the final product. The drink is rich in pre as well as probiotics and is a centuries old traditional method of improving gut health. What's more is that it packs a sour punch with blast of mustard, a flavor profile loved by the Indian palette since ages.

Quantitative Analysis

Rate-Mix Analysis

We will attack the problem by first breaking down both of our metrics on the basis of different segment of users. This will give us the efficiency (Rate) measures for each separate population bracket (Mix). Further we will use the Rate-Mix framework to do a Root Cause Analysis. This will allow us to break down the metric to an atomic level and help us attribute responsibility and identify where we need to focus the most.

In this data-sheet we do Rate-Mix Analysis for two levers - CTR and CAC.

In this project, we will focus on CTR analysis.

Insights from the Bar-Graphs:

- The impact of Rate dropping is visible across every cut of the data.

- We can address this by looking factors like - Quality of New Advertisements, Recent Page Content and Overall Changes in User Engagement

- The impact of Mix varied across Tiers and Zones - to further understand what is happening we further break-down the data

Comparison
Split	Mix Impact	Rate Impact
Tier 1 - North Zone	-1.62	-0.18
Tier 2 - North Zone	1.56	-0.72
Tier 3 - North Zone	0.77	-0.36
Tier 1 - East Zone	-3.20	-0.36
Tier 2 - East Zone	1.70	-0.84
Tier 3 - East Zone	-0.85	-0.35
Tier 1 - West Zone	0.80	-0.63
Tier 2 - West Zone	-1.62	-0.48
Tier 3 - West Zone	3.00	-0.06
Tier 1 - South Zone	-0.80	0.12
Tier 2 - South Zone	-1.68	-0.06
Tier 3 - South Zone	1.54	-0.21
Total	-0.40	-4.13

Insights from the Table:

- As seen above, the Impact of Rate dropping is clearly visible and is 10x than the impact of Mix

- There are certain Tier-Zone combinations where we see outlier - Tier 1 East Zone (-3.2%), Tier 3 West Zone (+3%)

- An insight than can be drawn from here is that the new content strategy is very polarizing.

- We double down on the new content strategy in the geographies it is working in and create and experiment with new content strategies in the geographies it is not

FUGLY

The issue that we want to diagnose is captured in the following table:

	Sept'24	Oct'24
CTR	0.81	0.76
Land to Add-To-Cart Rate	0.45	0.37
Add-To-Cart to Check-Out Rate	0.82	0.81

Effective Conversion Rate	0.30	0.23
	Delta	-0.07
	Drop	-23.47

Here we can see that from September to October the Effective Conversion Rate has dropped by 23%. If we break down the number into its funnel components we observe two things:

- Bottom of the Funnel is not affected significantly - The rate of Add-To-Cart to Purchase is steady at around 80%

- Top of the Funnel drop has been accounted for - Though there is a drop of 5% in the ToFu rate, we have strategies in place to address it

-Middle of the Funnel is affected severely - We see a sharp drop of 8% in the MoFu rate. This is what we will diagnose further

We start with breaking down each visit to the site into the parameters and flags we have access for - Zone, Tier, Age-Bracket, Add-To-Cart, Check-Out. We do two things - a frequency analysis and propensity analysis on each profile column - Zone, Tier and Age-Bracket. From this find out that the metrics for Zone and Tier are within tolerable bounds between two months but Age-Bracket is the outlier.

The following table showcases this:

Sept'24	Oct'24
Age-Bracket	Representation	Land to A2C	Age-Bracket	Representation	Land to A2C
18-25	0.10	0.07	18-25	0.20	0.11
25-30	0.45	0.48	25-30	0.40	0.42
30-40	0.35	0.52	30-40	0.30	0.46
40+	0.10	0.51	40+	0.10	0.47

We can drive the following insights from this table:

- From September to October we see an uptick of 2x in the 18-25 age bracket.

- The purchase propensity of 18-25 age bracket is the least amongst all

Linking this back to the New Content Strategy we further find out that the new content is performing well with the 18-25 age-bracket but not doing great in other age-brackets. We can now create experiments which tests-out how our content strategy can be optimized for the age-brackets which have higher purchase propensity.

Experimentation

Overview

We have understood that the New Content Strategy is not working from the RMA and FUGLY analysis done above. We now design experiments around our content strategy to find test out the following hypothesis - "Different content strategies for different age-brackets will help increase the CTR".

We define a success criteria and metric for the experiment to be deemed successful - the CTR should increase by 5% from October 2024 baseline of 76%.

Design of Experiment

A 5% absolute change on 76% translates to a relative change of around 6.5%, using a cohort size calculator we find that to prove our result with a confidence of 95% we need a cohort size of 790.

As we only have access to number of impressions we will consider the data for 800 impressions. An impression is defined as an individual seeing your advertisement once.

Cohort characteristics:

Variation A: This cohort is made up of 800 impressions of the old version of our advertisement. The cohort is set-up in such a way that each age-bracket is equally represented.

Variation B: The demographic composition of this cohort is kept similar to Variation A, in this variation each age-bracket is shown a different tailored version the advertisement.

Setup: To counter any seasonality and cyclicity effect we will run Variation A and Variation B of our experiment on alternate days starting on Monday and ending it after 14-Days.

Results

- After running the experiment for a period of 14 days we see a trend which aligns with our hypotheses as shown in the table below

- Though the trend is in favor of the hypotheses, the change observed is not enough to declare statistical significance yet, therefore we take a call to run the experiment for 14 more days.